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3.1.3.1 IDW-S Core Subsystem Data and Data Processing 

The IDW-S Core Subsystem holds and processes data from the following approved data sources: 
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• Violent Gang and Terrorist Organization File (V GTOF) from the Criminal Justice 
Information Systems (CJIS) Division 

• Open Source News 
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3.5.1 Data Processed 
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3.5.1.1 IDW-S Core Subsystem Data Processed 

Table 3. 5. 1.1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 
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3.5.1. 1 IDW-S Core Subsystem Data Processed 

Table 3.5. 1.1 below outlines the LDW-S datasets and information about transfer and ingest 
processing for each source. 
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Automated Transfer of Raw Data to D>W-S 

Raw data is automatically transferred to the IDW-S system from approved data sources in 
accordance with an Interconnection Agreement, Memorandum of Understanding, Memorandum 
of Agreement or Interface Control Document. IDW-S VI receives update data by the following 
means: 

• ACS update data is transmitted to the IDW-S system! 

• IntelPlus update data is transmitted to the IDW-S system! 

• SAMNet update data is transmitted to the 3DW-S system/ 


VGTOF data is transmitted to the IDW-S system \ 

Open Source News data is transmitted to the IDW^S system) 
The JICI library is static and is not updated on IDW-S VI. 



UNCLASSIFIED // FOR OFFICIAL USE ONLY 




UNCLASSIFIED // FOR OFFICIAL USE ONLY 

Investigative Data Warehouse - Secret (IDW-S) 

System Security Plan 
25 January 2005 
Version 1.6 



/ data into individual serial records (in the case of ACS ECF), 

document s (IntelPlus, JICI, Open Source News), messages (SAMNetL or files 
(VGTOF)I 



UNCLASSIFIED // FOR OFFICIAL USE ONLY 





UNCLASSIFIED // FOR OFFICIAL USE ONLY 

Investigative Data Warehouse - Secret (IDW-S) 

System Security Plan 
25 January 2005 
Version 1.6 



3.5.1 Data Processed 


Outside the Scope 


3.5.1.1 IDW-S Core Subsystem Data Processed 

Table 3. 5. 1.1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 
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INVESTIGATIVE DATA WAREHOUSE 


User Reference for 
IDW Version 1.1 



IDW Training Office 
Federal Bureau of Investigation 
Department of Justice 
935 Pennsylvania Street • Room 10184 
Washington, D.C. 20535 
Phone 202.324.2606 



ACS: Automated Case Support (CT Case Files) Archived Investigative dating back through calendar year 
1995. Updates pushed (FTP) to IDW once per day. 

2. Intel Plus: Updates pushed (FTP) to IDW as needed (us ually once per day). 



3* JICI: Archived data from the Joint Intelligence Committee Investigation (JICI) regarding 9/1 1 as directed by 
the Congress of the United States. (Static file system) 

ORCON* Parsed data from SAMNet and ACS data collections with this caveat. Pushed to IDW once 
per day. 

SAMNet: SECRET and below cable traffic to/ from FBI Headquarters. Pulled (FTP) from SAMNet 
server (IDW chronological job runs every 10 minutes; SAMNet posts new data 3 times per day). 

TFRG: Terrorist Financial records. Updates pushed (FTP) to IDW once per day. 

VGTOF: Violent Gang & Terrorist Offense File. Updated data file is provided in entirety by CD on a 
weekly basis. 

8. OSINT: Open Source Research (OSINT): Current from the MiTAP server. 
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INVESTIGATIVE DATA WAREHOUSE 



Hi 


- SECRET 


SYSTEM SECURITY PLAN 
INTRODUCTION 

The Federal Bureau of Investigation’s (FBI) Investigative Data Warehouse (IDW-S) is an 
initial data warehouse, content management and data mining system that will permit FBI 
investigative, analytical, administrative and intelligence personnel to access aggregated 
data previously only available through individual applications. The IDW-S system will be 
authorized to process classified national security data up to, and inclu din g, Secret. The 
IDW-S system is the successor of the Secure Counter-Terrorism/Collaboration Operational 
Prototype Environment (SCOPE”). 



Data processed by the system will include the following data sets: 


Outside the Scope 


• Approved case files from the FBI’s Automated Case Support (ACS) case 
management system; 

• Electronic versions of the Joint Intelligence Committee Investigation 
(JICI) defined archived documents; 
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• Secure Automated Messaging Network (SAMNet) message traffic; 

• IntelPlus File Rooms; and 

• Violent Gang and Terrorist Organization File (VGTOF”) from the 
Criminal Justice Information Systems (CJIS) Division; 


• Defense Advanced Research Projects Agency (DARPA) Translingual 
Information Detection, Extraction and Summarization (TIDES) Open 
Source Data 
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Data Sources Outside the Scope 


IDW-S contains data from the following data sources with security classification indicated 
in brackets: 


• Automated Case Support (ACS) System, Electronic Case File (ECF) 
Subsystem [Secret and below] 

• IntelPlus [Secret and below] 

• Secure Automated Message Network - Secret (SAMNet-S) [Secret and 
below] 

• NCIC Violent Gang and Terrorist Offender File (VGTOF) [Sensitive But 
Unclassified] 

• Joint Intelligence Committee Investigation (JICI) [Secret and below] 



- Received dat 


nto individual serial records (in the case of ACS 


ECF), documents (IntelPlus, JICI, Open Source News), messages 
(SAMNet), or files (VGTOF). 
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3.5. Data Processed 

Outside the Scope 

IDW-S processes the following data: 

3.5.1 ACS ECF 

IDW-S contains a subset of the ECF (Electronic Case File) subsystem of the 
Automated Case System (ACS). This subset consists of serials in those case 
classifications/subclassifications that have been officially sanctioned for inclusion 
in IDW. For each such serial, the ECF data includes metadata and text. IDW-S is 
synchronized against the ECF system once a day, a process which consists of 

receiving and pr ocessing the previous day’s increment of ADD MOD and 

DELETE records! 


3.5.2 IntelPlus b7E 

IDW-S curre ntly contains several Inte lPlus counter-terrorism (CT) Filerooms: 
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3.5.3 SAMNet 


b2 

b7E 


The Secure Automated Messaging N etwork ( SAMNet) data source consists of ca ble 
traffic messages received by the FBI J 

\ SAMNet provides only ADD record 

types, and SAMNet data is updated in IDW-S three times a day. SAMNet d a d 

3.5.4 VGTOF ~ ^ 


b2 

b7E 


Violent Gang Terrorist Offender File (VGTOF) data is provided by the FBI 
National Crime Information Center (NCIC). VGTOF data includes two 
components: Data/metadata for each named individual/offender and potentially 
multiple JPEG images per individual/ ~ ~ 


b2 

b7E 
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3.5.5 JICI 


The Joint Intelligence Committee Investigation (JICI) data collection was created following 
the September 11 attacks. Paper counter-terrorism files/folders from all FBI Field Offices 



3.5.6 Open Source News 


to 2 
to7E 


The IDW-S VI. 0 system contains an Open Source News library collected by the 
DARPA TIDES Program. These are primarily news source from around the world 
that are either in English or have been translated into English. Open Source News 
-d ata is re ceived in the form of text files (one per news article)} 


News data if 


JThe Open Source News data goes into IDW-S once a day. Open Source 


Open Source News material is derived from the following sources: 


Addis Ababa Tribune - http://www.addistribune.com/ 

Agencia Brasilia - http://www.radiobras.gov.br/ 

Al-Ahram (Egypt, weekly English version) - http://weekly.ahram.org.eg/ 
AllAfrica.com - http://allafrica.com/ 

Arabic News - http://www.arabicnews.com/ansub/ 

Asahi Shimbun - http://www.asahi.com/english/ 

Asia Times (Hong Kong) - http://www.atimes.com/ 

Bangkok Post - http://www.bangkokpost.net/ 

Christian Science Monitor - http://www.csmonitor.com/ 

Crescent International - http://www.muslimedia.com/mainpage.htm 
Daily Telegraph (London, England) - http://news.telegraph.co.xjk/ 

Dawn (Karachi, Pakistan) - http://www.dawn.com/ 

Debka (Israel) - http://www.debka.com 

East Africa Daily Nation - www.nationaudio.com/News/DailyNation/Today/ 
Gulf News (UAE) - http://www.gulf-news.com/ 

Ha'aretz (Israel) - http://www.haaretzdaily.com/ 

IFRC Intemation Federation of the Red Cross - http://www.ifrc.org/ 

IRIN Integrated Regional Information Network - http://www.irinnews.org/ 
Iraq Press News Agency - http://www.iraqpress.org/ 
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Islamic Republic News Agency (Iran) - http://www.ima.com/en 
Jakarta Post - http://www.thejakartapost.com/ 

Janes - http://www.janes.com/ 

Jordan Times - http://www.jordantimes.com/ 

L'Osservatore Romano - www.vatican.va/news_services/or/or_eng/text.html 
Lagos (Nigeria) Guardian - http://www.guardiannewsngr.com/ 

Lahore (Pakistan) Nation - http://www.nation.com.pk/ 

Lebanon Daily Star (Beirut) - http://www.dailystar.com.lb/ 

Malaysian Star - http://thestar.com.my/ 

Manila Bulletin - http://www.mb.com.ph/ 

Manila Times - http://www.manilatimes.net/ 

Miami Herald - http://www.miami.com/ 

Moscow Times - http://www.themoscowtimes.com/ 

National Post and CP - http://www.canada.com/ 

New Straits Times (Kuala Lumpur) - http://www.emedia.com.my/Current_News/NST/ 
PETRA (Jordanian News Agency) - http://www.petra.gov.jo 
Pakistan Observer (Islamabad) - http://pakobserver.net/ 

Palestine Chronicle - http://palestinechronicle.com/ 

People's Daily (China) - http://english.peopledaily.com.cn/ 

Philippine Star - http://www.philstar.com/p bils tar/ 

Pravda - http://english.pravda.ru/ 

ProMed - epidemiology mailing list 

Russian Information Agency Novosti - http://en.rian.ru/ 

Russian Issues (Misc. Russian news) - http://www.therussianissues.com/ 

Russian Observer - http://www.russianobserver.com/ 

SABA (The News Agency of Yemen) - http://www.sabanews.gov.ye 
Saudi Gazette - http://www.saudigazette.com.sa/sgazette/ 

South African Dispatch - http://www.dispatch.co.za/ 

Sydney Morning Herald - http://www.smh.com.au 
Tehran Times - http://www.tehrantimes.com/ 

Times of India - http://timesofindia.indiatimes.com/cms.dll 
UNHCR UN High Commissioner on Refugees - http://www.unhcr.ch/ 

Ummah News - http://www.ummahnews.com/ 

Uzbekistan Report - http://www.uzreport.com/eng/ 

Washingtion Post - http://www.washingtonpost.com/ 

XinHua News Service - http://www.xinhuanet.com/english/ 

Yemen Times (weekly) - http://www.yementimes.com/ 

Yomiuri Shimbun (Japan) - http://www.yomiuri.co.jp/ 


3.5.7 Summary of Data Sources 





1 



!83@as§® 
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liESiBSSJBrra 


HHE 

ACS 

FBI Automated Case 
System (ACS) 

FTP 

IntelPlus 

FBI Intel Plus 

FTP 

SAMNet 

FBI Secure Automated 
Messaging Network 
(SAMNet) 

FTP 

VGTOF 

FBI National Crime 
Information Center (NCIC) 

CD-ROM 

JICI 

FBI Records Management 
Division (RMD), 
Document Laboratory 
(DocLab), FBIHQ 

Multiple 

Open 

Source 

News 

Translingual Information 
Detection, Extraction and 
Summarization (TIDES) 
Program, Defense Advanced 
lesearch Projects Agency 
(DARPA) 

CD-ROM 


Outside the Scope 
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4.1.3 SAMNet Files 

4.1.4 IntelPlus Files. 

4.1.5 JICI Files 


4.1.6 MiTAP Files.. 

4.1.7 VGTOF Files 

4.1.8 Doc Lab Files 
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Outside the Scope 


4 Custom Software Applications 
4.1 IDW-S Data Ingest Applications 

Figure 4.1.1 is the dataflow diagram for the IDW-S data ingest process. 



Figure 4.1.1 IDW-S Data Ingest Dataflow 


Outside the Scope 
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Outside the Scope 


Data ingest processing on IDW-S includes the following: 
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• SAM Net Processing: The SAMNet data consists of cable traffic messages received by the 


FBI. 


: ~ — ■ — i 


1 

I 


JICI Processing: The JICI data consists of one file room where each case file and serial is 
represent ed by a directory. Each directory contains the text and image files associated with 
the serial / \ 

1 JICI data is provided in 


entirety. 

JVIiTAP Processing: The MiTAP data c onsists of file archives of news messages. 


VC TOF Processing: VGTOF is a structured data source that is currently available in IDW 
V 1 4 1 The VGTOF data is 

composed of an ASCII source file of structured data records ancf 


b2 

b7E 


files. The records for each individual are com bined into a document.! 


the previous data. 


JVGTOF data is provided in entirety, replacing 

Outside the Scope 
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4.1 Operations 

The IDW system is broken up into three subsystems, each with a different set of responsibilities 
DOCLAB System 

IDW-S contains data from the following data sources with security classification indicated in the 
table below: 


Nolc: . lardcopy versions of this document must be verified for correct version number with the IDW CM Document Control Library. 
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IDW Operational Services 
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FBI Automated 
Case System 
(ACS) 


Secret and below 


FBI IntelPlus 

Secret and below 

FBI Secure 

Automated 

Messaging 

Network 

(SAMNet) 

Secret and below 

FBI National 
Crime Information 
Center (NC1C) 

SBU 

FBI Records 
Management 
Division (RMD), 
Document 
Laboratory 
(DocLab), FBIHQ 

Secret and below 

MiTAP 

Unclassified 


b2 
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Note: Hardcopy versions of ttiis document must be verified for correct version number with the IDW CM Document Control Library. 
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The SPT-specific data sources will include: 



Unified Name Index (UNI) extracts 


Outside the Scope 


b2 
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IDW-S is an FBI enterprise program/system and provides data and data processing/analysis 
services to FBI agents and analysts as they perform counter-terrorism, counter-intelligence, and 
law enforcement missions. The initial focus for the IDW-S Core Subsystem has been to support 
data analysis/data mining for the following FBI groups/activities: 

Counter-Terrorism Division (CTD) 

Special Event Unit 

In addition, scanning/OCR performed in the DOCLAB-S subsystem supports the following FBI 
projects/programs: 


■Outside the Scope 



Joint Intelligence 
Committee 
Investigation (JICI) 


IntelPlus 


Secret 


Secret 


Division 
Federal Bureau of Investigation 
J. Edgar Hoover FBI Building 
935 Pennsylvania Avenue, NW 
Washington, DC 20535 


Information Resources Division 
Federal Bureau of Investigation 
J. Edgar Hoover FBI Building 
935 Penasylvania Avenue, NW 
Washington, DC 20535 


b6 

b7C 
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Automated Transfer of Raw Data to IDW-S 


Raw data is automatically transferred to the IDW-S system from approved data sources in 
accordance with an Interconnection Agreement, Memorandum of Understanding, Memorandum 
of Agreement or Interface Control Document. IDW-S VI receives update data by the following 
means: 


ACS update data is transmitted to the IDW-S systerrl ~ 
IntelPlus update data is transmitted to the IDW-S svsterH 


SAMNet update data is transmitted to the IDW-S svstenl 


VGTOF data is transmitted to the IDW-S system! 


Open Source News data is transmitted to the IDW-S systenT 
The JICI library is static and is not updated on IDW-S VI. 


Outside the Scope 


b2 

b7E 
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Parse and Format (Concatenated) Data to Individual Files (HTML) 


- Initial processing of raw data file£ 


Jlata into individual serial records (in the case of ACS ECF), 


document s (IntelPlus, J1CI, Open Source News'), messages (SAMNetV or files 
(VGTOF )/ 


b2 

b7E 
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Outside the Scope 


3.5.1. 1 IDVV-S Core Subsystem Data Processed 

Table 3.5. 1 . 1 below outlines the IDW-S datasets and information about transfer and ingest 
processing for each source. 


b2 

b7E 


!■ 

HUM! 


ACS 

Electronic 
Case Files 
(EOF) 

FBI Automated 
Case System 
(ACS) 

Secret and below 
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IntelPlus 

Filerooms 

FBI IntelPlus 

Secret and below 

SAMNet 

FBI Secure 

Automated 

Messaging 

Network 

(SAMNet) 

Secret and below 

VGTOF 

■ 1 II llll III— 


Crime Information 
Center (NCIC) 

SBU 

JICI 

FBI Records 
Management 
Division (RMD), 
Document 
Laboratory 
(DocLab), FBIHQ 

Secret and below 

Open 

Source 

News 

MiTAP 

Unclassified 


Table 3.5.1. 1 IDW-S Da la Seta 


b2 
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3.9.9 Indirect Connections 


3.9.9.1 IDW-S Core Subsystem Indirect Connections 


l ease System 
(ACS) 

Secret 

FBI 

IP 

mem 

mm 


FBI Intel Plus 

Secret 

FBI 

FTP 




FBI Secure 

Automated 

Messaging 

Network 

(SAMNet) 

Secret 

FBI 

FTP 



b2 

blE 

J~B1 National 

Unclassified/SBU 

FBI 

CD-ROM 




Crime 







Information 
Center (NCIC) 







FBI Document 

Conversion 

Laboratory 

(DocLab). 

Records 

Management 

Division (RMD), 

FB1HQ 

Secret 

FBI 

CD/DVD 
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[MB 

IMM 




MiTAP - Open 
Source News 

Unclassified 

NA 

CD-ROM 



3.9.9.2 SPT Subsystem Indirect Connections 
None. 


3.9.9.3 DOCLAB-S Subsystem Indirect Connections 
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OTHER 0/S 


I DU, Version 1.0 is scheduled for release January 2004 
and will provide access to six primary data sources: 

IDU Version 1 .0 

o Electronic Case File (EC F) /Automated Case Support (ACS) 
/Virtual Case File - FBI investigative and intelligence 
information. 

o SAMNET - Hire traffic to the FBI from members of the 
intelligence community. 

o JICI - counterterrorism files that were scanned into a 
database to accommodate the Joint Intelligence 
Committee Investigation (JICI) on the September 
11, 2001, terrorist attack on the World Trade 
Center. 


b2 

blE 


o VGTOF - Violent Crime and Terrorist offender File 

information that includes biographical data 
pertaining to members of the identified groups. 

o Open Source - Fifty-seven neus sources from around the 
Horld that are either in English or have 
been translated into English. 






